Skip to content

feat(benchmark): add LoCoMo V2 evidence chain and report - #34

Merged
Peefy merged 1 commit into
mainfrom
feat/locomo-evidence-chain
Sep 10, 2026
Merged

feat(benchmark): add LoCoMo V2 evidence chain and report#34
Peefy merged 1 commit into
mainfrom
feat/locomo-evidence-chain

Conversation

@hrw991009

Copy link
Copy Markdown
Collaborator

Summary

  • add a LoCoMo V2 MiniCPM evaluation path with validated dataset mapping and dialog-mode RawMessage ingestion
  • keep chunking, indexing, retrieval, fusion, and reranking daemon-owned; persist ingestion, final retrieval, Answerer, Judge, and diagnostic evidence
  • add retry-safe checkpoints, preflight validation, a daemon launcher, and explicit execution-error accounting
  • add the completed LoCoMo V2 evaluation report and links to result, trace, session-ingest, and manifest artifacts

Evaluation result

  • completed: 1,492 / 1,492 questions
  • final execution errors: 0
  • LLM-judge accuracy: 931 / 1,492 (62.40%)
  • answerer: openrouter:deepseek/deepseek-v4-flash-0731
  • judge: openrouter:qwen/qwen3.7-flash
  • retrieval: daemon-default Top-12, dialog mode

LoCoMo V2 MiniCPM has no QA-level gold evidence IDs, so Recall@K, Hit@K, Precision@K, and MRR remain null rather than being inferred from session or parent metadata.

Validation

  • pnpm --filter @melandlabs/benchmark-locomo typecheck
  • pnpm --filter @melandlabs/benchmark-locomo test — 24 tests passed
  • pnpm exec prettier --check benchmark/locomo/docs/locomo-v2-test-report.md
  • source-scoped Biome lint completed with no errors; the smoke CLI retains 12 expected noConsole warnings
  • full 1,492-question run completed with result, trace, session-ingest, and manifest artifacts

Scope note

The final artifact used compatible checkpoint recovery and was produced from a dirty worktree. It is a complete end-to-end diagnostic result, not a clean fresh-database --no-resume baseline for leaderboard or version-to-version comparison.

@Peefy
Peefy merged commit bb12aea into main Sep 10, 2026
13 checks passed
@Peefy
Peefy deleted the feat/locomo-evidence-chain branch September 10, 2026 05:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants